Systems and methods for mitigating vigilance decrement while maintaining readiness using augmented reality in a vehicle

ABSTRACT

System, methods, and other embodiments described herein relate to improving vigilance and readiness of an operator in a vehicle that includes an augmented reality (AR) display. In one embodiment, a method includes computing an engagement level of the operator to characterize an extent of vigilance decrement and readiness presently exhibited by the operator relative to operating characteristics of the vehicle including an external environment around the vehicle and at least semi-autonomous operation of the vehicle. The method includes dynamically rendering, on the AR display, graphical elements by varying visual characteristics of the at least one graphical element as a function of the engagement level and based, at least in part, on sensor data about the external environment. Dynamically rendering the at least one graphical element improves the at least semi-autonomous operation of the vehicle through inducing vigilance and readiness within the operator with respect to the operating characteristics.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. Non-Provisionalapplication Ser. No. 15/586,347, filed on, May 4, 2017 now U.S. Pat. No.9,904,287, which is herein incorporated by reference in its entirety.

TECHNICAL FIELD

The subject matter described herein relates in general to systems andmethods for monitoring and preventing/mitigating vigilance decrementand, more particularly, to using augmented reality (AR) within a vehicleto maintain engagement of a vehicle operator.

BACKGROUND

Autonomous vehicles can sense a surrounding environment (e.g.,obstacles, roadway, etc.) and navigate autonomously through thesurrounding environment without human input or at least partiallywithout human input. That is, autonomous vehicles can operate accordingto different levels of autonomy. For example, an autonomous vehicle canoperate according to the Society of Automotive Engineers (SAE) Level 2classification for autonomous driving. In this autonomous operatingmode, an autonomous vehicle relies, at least in part, on handovers fromautonomous operation to manual control by a vehicle operator when thevehicle cannot or should not autonomously operate due to variouscircumstances.

However, whether initiated by the vehicle or through intervention froman operator, handovers can occur without much notice. Therefore, toreliably accomplish handovers, an operator generally should maintainboth readiness to act, should the autonomy disengage, and vigilance tosupervise and selectively take action when the autonomy of the vehiclefails to detect a dangerous situation.

Ensuring readiness and vigilance of the operator presents variousdifficulties. For example, because the vehicle can operate for extendedperiods without manual input from the operator under level 2 autonomy,operators can stop paying attention to the environment around thevehicle and begin daydreaming, engaging in unrelated tasks, and so on.Moreover, approaches such as audible alerts can be an annoyance to theoperator instead of an effective guarantor of engagement. Furthermore,approaches that require an operator to keep their hands on the steeringwheel and/or to maintain their eyes forward-facing can also beineffective since the operator may engage in the noted postures tosimply satisfy the monitoring system while still daydreaming or focusingon other tasks. Consequently, the noted approaches fail to maintain bothreadiness and vigilance of a vehicle operator.

SUMMARY

An example of a vigilance system for mitigating vigilance decrement ispresented herein. Vigilance decrement, in the context of operating avehicle, generally refers to a tendency of a vehicle operator to becomedisengaged from a surrounding environment over time when not physicallyengaged and, thus, less aware of aspects surrounding the vehicle andaspects that possibly affect the operation of the vehicle when thevehicle is operating autonomously. Accordingly, in one embodiment, thevigilance system is implemented in a vehicle that operates autonomously(e.g., SAE level 2) to facilitate keeping the operator engaged withaspects of the driving task and the surrounding environment. Moreover,the vigilance system is, for example, implemented within the vehiclealong with further systems such as operator monitoring systems (e.g.,cameras for eye-tracking), an augmented reality (AR) system, and so on.Therefore, in one embodiment, the vigilance system simultaneouslymonitors an operator of the vehicle and controls an augmented realitydisplay to render different visuals that induce the operator to be bothvigilant of a possible need to take control from the autonomous system,and ready to control the vehicle should the autonomous systems execute ahandover to manual control.

In one embodiment, a vigilance system for improving vigilance andreadiness of an operator in a vehicle that includes an augmented reality(AR) display is disclosed. The system includes one or more processorsand a memory that is communicably coupled to the one or more processors.The memory stores a vigilance module that includes instructions thatwhen executed by the one or more processors cause the one or moreprocessors to compute an engagement level of the operator tocharacterize an extent of vigilance decrement and readiness presentlyexhibited by the operator relative to operating characteristics of thevehicle including an external environment around the vehicle and atleast semi-autonomous operation of the vehicle. The memory stores arendering module including instructions that when executed by the one ormore processors cause the one or more processors to dynamically render,on the AR display, at least one graphical element by varying visualcharacteristics of the at least one graphical element as a function ofthe engagement level and based, at least in part, on sensor data aboutthe external environment. The rendering module includes instructions todynamically render the at least one graphical element to improve the atleast semi-autonomous operation of the vehicle through inducingvigilance and readiness within the operator with respect to theoperating characteristics.

In one embodiment, a non-transitory computer-readable medium forimproving vigilance and readiness of an operator in a vehicle thatincludes an augmented reality (AR) display is disclosed. Thecomputer-readable medium stores instructions that when executed by oneor more processors cause the one or more processors to perform thedisclosed functions. The instructions include instructions to compute anengagement level of the operator to characterize an extent of vigilancedecrement and readiness presently exhibited by the operator relative tooperating characteristics of the vehicle including an externalenvironment around the vehicle and at least semi-autonomous operation ofthe vehicle. The instructions include instructions to dynamicallyrender, on the AR display, at least one graphical element by varyingvisual characteristics of the at least one graphical element as afunction of the engagement level and based, at least in part, on sensordata about the external environment. The instructions dynamically renderthe at least one graphical element improve the at least semi-autonomousoperation of the vehicle through inducing vigilance and readiness withinthe operator with respect to the operating characteristics.

In one embodiment, a method of improving vigilance and readiness of anoperator in a vehicle that includes an augmented reality (AR) display isdisclosed. The method includes computing an engagement level of theoperator to characterize an extent of vigilance decrement and readinesspresently exhibited by the operator relative to operatingcharacteristics of the vehicle including an external environment aroundthe vehicle and at least semi-autonomous operation of the vehicle. Themethod includes dynamically rendering, on the AR display, at least onegraphical element by varying visual characteristics of the at least onegraphical element as a function of the engagement level and based, atleast in part, on sensor data about the external environment.Dynamically rendering the at least one graphical element improves the atleast semi-autonomous operation of the vehicle through inducingvigilance and readiness within the operator with respect to theoperating characteristics.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate various systems, methods, andother embodiments of the disclosure. It will be appreciated that theillustrated element boundaries (e.g., boxes, groups of boxes, or othershapes) in the figures represent one embodiment of the boundaries. Insome embodiments, one element may be designed as multiple elements ormultiple elements may be designed as one element. In some embodiments,an element shown as an internal component of another element may beimplemented as an external component and vice versa. Furthermore,elements may not be drawn to scale.

FIG. 1 illustrates one embodiment of a vehicle within which systems andmethods disclosed herein may be implemented.

FIG. 2 illustrates one embodiment of a vigilance system that isassociated with inducing an operator of a vehicle to be engaged withdriving tasks.

FIG. 3 illustrates a schematic of a feedback loop between elements ofthe vigilance system of FIG. 2.

FIG. 4 illustrates one embodiment of a method that is associated withrendering elements within an AR system to engage a vehicle operator andthereby maintain vigilance and readiness of the vehicle operator.

FIG. 5 illustrates an interior view of a vehicle and elements displayedwithin an AR system.

FIG. 6 illustrates an interior view of a vehicle that is similar to theview of FIG. 5 but is illustrated with further elements displayed withinan AR system.

FIG. 7 illustrates an interior view of a vehicle and elements displayedwithin an AR system.

FIG. 8 illustrates an interior view of a vehicle that is similar to theview of FIG. 7 but is illustrated with further elements to engage anoperator with vigilance decrement.

FIG. 9 illustrates an interior view of a vehicle and elements displayedwithin an AR system.

FIG. 10 illustrates an aerial view of a vehicle and elements within asurrounding environment as rendered for display within an AR system.

DETAILED DESCRIPTION

Systems, methods and other embodiments associated with mitigatingvigilance decrement and assuring readiness within a vehicle operator byusing an augmented reality (AR) display to engage the vehicle operatorare disclosed. As mentioned previously, an operator can becomedisengaged from surroundings of the vehicle and aspects relating to theoperation of the vehicle when the vehicle operates autonomously. Thus,the operator can suffer from vigilance decrement as the vehiclecontinues to operate autonomously since the operator need not be engagedwith what the vehicle is doing for the vehicle to continue operating. Ingeneral, the vigilance decrement of the operator is a deterioration inthe ability to remain vigilant (e.g., alert, watchful, etc.) for eventsoccurring around the vehicle as time progresses. Thus, within thecontext of an autonomous vehicle, the vigilance decrement relates to atendency of a vehicle operator to become disengaged from operating thevehicle over time when not actively controlling/operating the vehicle.Consequently, because of the vigilance decrement, the vehicle operatormay not be sufficiently engaged with the operation of the vehicle tomanually intervene and initiate a handover when the autonomy should notcontrol the vehicle. Similarly, when the vehicle is operatingautonomously and performs a handover to manual control, the vehicleoperator can lack the appropriate readiness to control the vehiclebecause of becoming disengaged from the driving tasks and the presentoperating environment of the vehicle.

Thus, in one embodiment, the vigilance system is implemented in avehicle that operates autonomously (e.g., SAE level 2) to facilitatekeeping the operator engaged with aspects of the driving task and thepresent operating environment. Moreover, the vigilance system is, forexample, implemented within the vehicle along with further systems suchas operator monitoring systems (e.g., cameras for eye-tracking), anaugmented reality (AR) system, and so on. In one embodiment, thevigilance system simultaneously monitors an operator of the vehicle,computes an engagement level of the operator, and controls an augmentedreality display to render different visuals. In this way, the vigilancesystem can induce the operator to be both vigilant to intervene with theoperation of the vehicle when judged appropriate by the operator for aparticular circumstance and ready to control the vehicle should theautonomous systems execute a handover to manual control. In other words,by providing an experience to the operator through the AR system that isgenerated as a function of a present engagement level of the operator,the vigilance system can facilitate preventing the operator fromexperiencing vigilance decrement and cause the operator to maintainengagement with the vehicle to improve a likelihood that the operator isvigilant and ready to control the vehicle.

Referring to FIG. 1, an example of a vehicle 100 is illustrated. As usedherein, a “vehicle” is any form of motorized transport. In one or moreimplementations, the vehicle 100 is an automobile. While arrangementswill be described herein with respect to automobiles, it will beunderstood that embodiments are not limited to automobiles. In someimplementations, the vehicle 100 may be any other form of motorizedtransport that, for example, can operate at least semi-autonomously,includes an augmented reality (AR) system or capabilities to support anAR system, and thus benefits from the functionality discussed herein.

The vehicle 100 also includes various elements. It will be understoodthat in various embodiments it may not be necessary for the vehicle 100to have all of the elements shown in FIG. 1. The vehicle 100 can haveany combination of the various elements shown in FIG. 1. Further, thevehicle 100 can have additional elements to those shown in FIG. 1. Insome arrangements, the vehicle 100 may be implemented without one ormore of the elements shown in FIG. 1. While the various elements areshown as being located within the vehicle 100 in FIG. 1, it will beunderstood that one or more of these elements can be located external tothe vehicle 100. Further, the elements shown may be physically separatedby large distances.

Some of the possible elements of the vehicle 100 are shown in FIG. 1 andwill be described along with subsequent figures. However, a descriptionof many of the elements in FIG. 1 will be provided after the discussionof FIGS. 2-10 for purposes of brevity of this description. Additionally,it will be appreciated that for simplicity and clarity of illustration,where appropriate, reference numerals have been repeated among thedifferent figures to indicate corresponding or analogous elements. Inaddition, the discussion outlines numerous specific details to provide athorough understanding of the embodiments described herein. Those ofskill in the art, however, will understand that the embodimentsdescribed herein may be practiced using various combinations of theseelements.

In either case, the vehicle 100 includes a vigilance system 170 that isimplemented to perform methods and other functions as disclosed hereinrelating to mitigating vigilance decrement and promoting engagement by avehicle operator. Additionally, an augmented reality system 180 isillustrated as an additional aspect of the vehicle 100. However, itshould be noted that while the AR system 180 is illustrated as asub-component of the vehicle 100, in various embodiments, the AR system180 can be partially integrated with the vehicle or separate from thevehicle 100. Thus, in one or more embodiments, the AR system 180 cancommunicate via a wired or wireless connection with the vehicle 100 tocorrelate functionality as discussed herein. Moreover, the AR system 180can include one or more displays (e.g., integrated or mobile) withinwhich to display graphic elements to the operator and/or passengers.

It should be appreciated that the AR system 180 can take many differentforms but in general functions to augment or otherwise supplementviewing of objects within a real-world environment surrounding thevehicle. That is, for example, the AR system 180 can overlay graphicsusing one or more AR displays in order to provide for an appearance thatthe graphics are integrated with the real-world through, for example,the windshield of the vehicle 100. Thus, the AR system 180 can includedisplays integrated with the windshield, side windows, rear windows,mirrors and other aspects of the vehicle 100. In further aspects, the ARsystem 180 can include head-mounted displays such as goggles or glasses.In either case, the AR system 180 functions to render graphical elementsthat are in addition to objects in the real-world, modifications ofobjects in the real-world, and/or a combination of the two. In oneembodiment, at least one AR display of the AR system 180 fuses areal-time image from a camera (e.g., exterior facing camera) of at leastpart of the surroundings of the vehicle 100 with synthetic objects(e.g., rendered graphical elements) from the AR system 180 and/or thevigilance system 170. As one example, a monitor (i.e., AR display) isintegrated within or just above a dashboard of the vehicle 100 and iscontrolled to display a fused view of graphical elements rendered by theAR system 180 with real-world images from the camera. In this way, theAR system 180 can augment or otherwise modify a view of anoperator/passenger in order to provide an enriched/embellished visualsensory experience. The noted functions and methods will become moreapparent with a further discussion of the figures.

With reference to FIG. 2, one embodiment of the vigilance system 170 ofFIG. 1 is further illustrated. The vigilance system 170 is shown asincluding a processor 110 from the vehicle 100 of FIG. 1. Accordingly,the processor 110 may be a part of the vigilance system 170, thevigilance system 170 may include a separate processor from the processor110 of the vehicle 100, or the vigilance system 170 may access theprocessor 110 through a data bus or another communication path. In oneembodiment, the vigilance system 170 includes a memory 210 that stores amonitoring module 220, a vigilance module 230, and a rendering module240. The memory 210 is a random-access memory (RAM), read-only memory(ROM), a hard-disk drive, a flash memory, or other suitable memory forstoring the modules 220, 230, and 240. The modules 220, 230, and 240are, for example, computer-readable instructions that when executed bythe processor 110, cause the processor 110 to perform the variousfunctions disclosed herein.

Accordingly, the monitoring module 220 generally includes instructionsthat function to control the processor 110 to retrieve data from sensorsof a sensor system 120 of the vehicle 100. In other words, themonitoring module 220 includes instructions to acquire operator stateinformation 260 that characterizes a present mental state of theoperator, present actions of the operator, where a gaze of the operatormay be directed, autonomic responses of the operator, biologicalresponses/conditions of the operator, and so on. It should beappreciated that the present disclosure provides an exemplary listing ofaspects associated with the operator that can be monitored to producethe operator state information 260; however, this listing is not to beconstrued as limiting and is provided as an exemplary list ofpossibilities for purposes of this discussion.

Accordingly, by way of example, the operator state information 260 caninclude information about a direction of a gaze, a path/track of thegaze, heart rate, blood pressure, respiratory function, blood oxygenlevels, perspiration levels, pupil dilation/size, brain activity (e.g.,EEG data), salivation information, hand/arm positions, foot/legpositions, a general orientation of the operator in the vehicle 100(e.g., forward-facing, rear-facing, etc.), seat position, rate ofmovement, facial feature movements (e.g., mouth, blinking eyes, movinghead, etc.), and so on.

Additionally, the monitoring module 220 can determine the operator stateinformation 260 in multiple different ways depending on a particularimplementation. In one embodiment, the monitoring module 220communicates with various sensors of the sensor system 120 including oneor more of: camera(s) 126 (e.g., for gaze/eye tracking), heart ratemonitor sensors, infrared sensors, seat position sensors, and so on. Inone embodiment, the sensors are located within a passenger compartmentof the vehicle 100 and can be positioned in various locations in orderto acquire information about the noted aspects of the operator and/oraspects related to the operator. Furthermore, the sensor system 120 caninclude multiple redundant ones of the sensors in order to, for example,improve accuracy/precision of collected operator state information 260.

With continued reference to the vigilance system 170, in one embodiment,the system 170 includes the database 250. The database 250 is, in oneembodiment, an electronic data structure stored in the memory 210 oranother data store and that is configured with routines that can beexecuted by the processor 110 for analyzing stored data, providingstored data, organizing stored data, and so on. Thus, in one embodiment,the database 250 stores data used by the modules 220, 230, and 240 inexecuting various functions. In one embodiment, the database 250includes the operator state information 260 collected by the monitoringmodule 220 in addition to further data used by the vigilance system 170such as the vigilance model 270.

The vigilance module 230, in one embodiment, generally includesinstructions that function to control the processor 110 to compute anengagement level from the operator state information 260. That is, thevigilance module 230 uses the operator state information 260 tocharacterize whether the operator is presently engaged with the vehicle100 and the present operating environment of the vehicle 100. Ingeneral, the engagement level is a quantization of an extent to whichthe operator is engaged with how the autonomous driving module 160 isautonomously controlling the vehicle 100 and is aware ofsituational/contextual information about the operation of the vehicle100. Accordingly, the vigilance module 230 can generate the engagementlevel as an assessment of a present mental state of the operator inrelation to whether the operator is engaged and aware of the operationof the vehicle 100 and, for example, may not indicate whether theoperator is physically ready/positioned to manually control the vehicle100. However, in further embodiments, the vigilance module 230 cancompute the engagement level to account for both the present mentalstate of the operator and a readiness of the operator to manuallycontrol the vehicle 100.

Thus, by way of a brief example, the vigilance module 230 can, forexample, electronically access the operator state information 260including information about a gaze of the operator. As such, thevigilance module 230 assesses the information about a gaze of theoperator and, more particularly, a duration for which the operator hasbeen gazing in a particular location/direction. From this assessment andaccording to, for example, the vigilance model 270, the vigilance module230 can characterize the engagement of the operator and whether theoperator is vigilant and ready to assume manual control of the vehicle100. For example, the vigilance module 230 considers whether the gaze isdirected toward the roadway in the front of the vehicle 100, whether thegaze is tracking objects around the vehicle 100, whether the gaze ischecking instrument readings, and so on. From this assessment, thevigilance module 230 uses the vigilance model 270 to generate theengagement level. Thus, the vigilance model 270 informs the vigilancemodule 230 about a likelihood of the operator being engaged according tothe various noted aspects of the gaze.

In further aspects, the vigilance module 230 can undertake a morecomplex analysis in order to compute the engagement level. For example,the vigilance module 230 can correlate a direction of the gaze withfurther information from the sensor system 120 such as apresence/location of different objects and a relation of the differentobjects (e.g., cars moving toward vs. away) to the vehicle 100.Additionally, the vigilance module 230 can further consider dynamicvehicle data when computing the engagement level. In one embodiment, thedynamic vehicle data can include various aspects of the vehicle 100 andexternal environmental aspects relating to the present operatingenvironment. For example, the dynamic vehicle data can include GPScoordinates of the vehicle 100, vehicle telematics, roadway types (e.g.,highway vs city), terrain characteristics (e.g., flat/plain surroundingsvs feature-filled), duration of travel, weather conditions, time of day,level of traffic, current speed, infotainment information (e.g., whethera radio is playing, etc.), temperature, number of passengers, and otherdata collected by the monitoring module 220 from different sensors andsystems of the vehicle 100. In general, reference to the presentoperating environment of the vehicle 100 as used herein relates toinformation about the environment surrounding the vehicle 100, objectsin the environment, and an effect of the operation of the vehicle 100 onthe environment and the objects.

Additionally, the vigilance module 230 can use data from various sensorsincluding the LIDAR 124, the radar 123, the cameras 126 (e.g., internaland external cameras), the sonar 125, navigation system 147, and so on.In one embodiment, the vigilance module 230 can further electronicallyaccess and use information from the autonomous driving module 160 abouttrajectories of objects in the present operating environment, potentialhazards, current decision state information associated with autonomousoperation/controls/path planning, and so on.

Thus, to compute the engagement level, the vigilance module 230, in oneembodiment, analyzes the operator state information 260 along with theadditional information (e.g., the dynamic vehicle data) to quantize anextent of engagement (e.g., mental state/vigilance and/or readiness) bythe operator. In general, the vigilance module 230 identifies thevarious data elements in relation to the vigilance model 270, whichprovides, for example, likelihoods of operator engagement according tothe separate data elements. Subsequently, the vigilance module 230 cancombine the individual likelihoods according to defined weightings toproduce an overall likelihood. Consequently, the vigilance module 230can provide the overall likelihood as the engagement level or applyfurther analysis using the overall likelihood so that the engagementlevel is further customized according to the present operatingenvironment of the vehicle 100 by, for example, translating thelikelihood into a particular classification (e.g., 0-10) according toaspects (e.g., a degree of skill/awareness needed to navigate currentsurroundings) of the present operating environment.

In further embodiments, the vigilance module 230 in combination with thevigilance model 270 form a computational model such as a machinelearning logic, deep learning logic, a neural network model, or anothersimilar approach. In either case, the vigilance module 230, whenimplemented as a neural network model or another computational model, inone embodiment, electronically processes the operator state information260 and other electronic data (e.g., dynamic vehicle data) as inputs tothe vigilance model 270. Accordingly, the vigilance module 230 inconcert with the vigilance model 270 produce the engagement level as anelectronic output that characterizes an extent of engagement of theoperator as, for example, a single electronic value.

It should be appreciated that while the engagement level is discussed asa single value, in general, the engagement level embodies both vigilanceof the operator in relation to supervising the autonomous operation ofthe vehicle 100 and readiness of the operator to control the vehicle 100in the event of the autonomous driving module 160 executing a handoverto manual control. Thus, in further implementations, the vigilancemodule 230 produces the engagement level with multiple electronic valuesthat characterize different aspects of the operator state (e.g., mentalstate and physical state) and the extent to which the operator ispresently engaged with driving tasks of the vehicle 100 and the presentoperating environment. For example, the engagement level can includesub-components identifying particular aspects of what the operator ispresently engaged with, an extent to which the operator is engaged withother devices/tasks, where the operator is most likely to gaze ifgraphics are rendered within the AR display, whether the operator isactively supervising operation of the vehicle 100 but not ready tomanually control the vehicle 100 (e.g., holding an ice cream cone ineach hand while watching operation of the vehicle 100), and so on.

Additionally, while the use of many different sources and forms of dataare described as being part of how the vigilance module 230 computes theengagement level, in various embodiments, the particular data can varyaccording to different aspects of the implementation. In any case, thepresent examples are intended for purposes of illustration and shouldnot be construed as a limiting form of the disclosed data elements andfunctions. Moreover, it should be appreciated that the discussedoperator state information 260 and additionally discussed electronicdata can be, in one embodiment, collected and used to train thevigilance model 270.

That is, the operator state information 260, the dynamic vehicle data,induced responses of the operator, and other data can be collected andstored while the vehicle 100 is operating. Thereafter, the vigilancesystem 170, in one embodiment, can communicate the collected informationto an OEM or another central collection point to be used in training themodel 270. In further embodiments, the vigilance system 170 can use thecollected information as a training data set to update or otherwisecustomize/improve the vigilance model 270 in relation to particularitiesof the operator of the vehicle 100. Thus, in one embodiment, thevigilance system 170 further trains the model 270 using the operatorstate information 260 and dynamic vehicle data as electronic inputs andby providing feedback to the model according to logged responses of theoperator and/or logged engagement levels computed by the vigilancemodule 230. In this way, the vigilance system can actively learn whichrenderings in the AR system 180 induce the operator to be vigilant andthus, adjust the displays accordingly.

Continuing with the discussion of the vigilance system 170 of FIG. 2,the rendering module 240, in one embodiment, generally includesinstructions that function to control the processor 110 to rendergraphical elements within a display of the AR system 180 as a functionof the engagement level. That is, the rendering module 240 reacts to howengaged the operator is with the present operating environment of thevehicle 100 by rendering graphical elements within the AR system 180 toinduce the operator to be engaged with the vehicle 100 and the presentoperating environment. For example, the rendering module 240 tailors thegraphical elements to engage the operator and induce the operator torespond by engaging the vehicle 100 and transitioning from distractedtasks to being situationally aware and ready to control the vehicle 100and/or intervene in control of the vehicle 100. Thus, the renderingmodule 240 can intensify/escalate how the graphical elements within theAR display are presented according to a particular engagement level. Inone embodiment, the rendering module 240 renders the graphical elementsby adjusting an opacity of the graphics, flashing the graphics orotherwise adjusting an animation, varying colors, rendering additionalgraphics, rendering arrows to direct the attention of the operator,rendering text including instructions and/or questions to the operator,rendering overlays on objects (e.g., cartoon animations, avatars, etc.),and so on.

In further aspects, the rendering module 240 can analyze or otherwiseelectronically mine information from other vehicle systems (e.g., thesensor system 120, navigation system 147, autonomous module 160) aboutthe vehicle 100 to determine aspects about the present operatingenvironment which can be related to the operator through the AR system180. The rendering module 240 uses the mined information to render thegraphical elements as active content that relates directly to thepresent operating environment of the vehicle 100. In other words, therendering module 240 renders the graphical elements to inform asituational awareness of the operator and to engage the operator withthe present operating environment. For example, the rendering module 240can render the graphical elements to depict and graphically identifyhazards, planned and/or projected driving paths, other vehicles,predicted movements of the other vehicles/objects, and so on.

In still further embodiments, the rendering module 240 renders complexgraphical overlays that illustrate visualizations of information aboutthe present operating environment as perceived via one or more sensors(e.g., LIDAR 124), from internal state information of the autonomousmodule 160, and so on. Thus, the rendering module 240 can produce animmersive visual environment through the AR system 180 that displaysgraphics about object tracks, current speeds and headings, predictedmaneuvers of the vehicle 100 and other vehicles, assessments of hazards,navigational path information, autonomous path planning and objectavoidance information, object detection and tracking, and so on. In oneexample, the rendering module 240 translates 3D point clouds produced bythe LIDAR 124 into graphical overlays of the present operatingenvironment such that objects perceived by the LIDAR 124 are overlaidwith data points from scans of the LIDAR 124. Similarly, the renderingmodule 240 can use data from volumetric scans by the radar 123 and/ordata from other sensors to produce animated/graphical overlays and addadditional information as graphical elements to visuals through by theAR system 180.

Additionally, the rendering module 240 can also render infotainmentgraphical elements including local points of interest, trivia relatingto a present location or nearby elements, and so on. In anotherembodiment, the rendering module 240 can additionally or alternativelyrender graphical elements within the present operating environmentthrough the AR system 180 as part of an interactive game with theoperator. For example, the rendering module 240 can render targets, gameicons, and other game style graphics. In one embodiment, the renderingmodule 240 renders graphics for a first-person interactive style ofgame, a real-time strategy game, and so on. Moreover, the renderingmodule 240, in one embodiment, controls audio within the vehicle 100 toprovide alerts, chimes, speech, music and other sounds in combinationwith rendering the graphical elements.

It should be appreciated that the rendering module 240 can produce manydifferent graphical elements within one or more displays (e.g., front,side, rear, head-mounted) of the AR system 180. However, regardless ofthe particular graphical elements, the rendering module 240 renders thegraphics as a function of how engaged the operator is with the presentoperation of the vehicle 100, as indicated by the engagement level.Thus, the rendering module 240 functions to dynamically generate contentwith the AR system 180 according to the engagement level to inducegreater vigilance and readiness within the operator and/or to maintainthe vigilance/readiness of the operator. Thus, the rendering module 240dynamically adjusts which graphical elements are rendered, how thegraphical elements are rendered, where the graphical elements arerendered, and so on in a manner that induces vigilance/readiness in theoperator and avoids difficulties associated with the vigilancedecrement.

As a further illustration of dynamic aspects relating to how thevigilance system 170 executes to adjust and update the AR system 180,the discussion will now transition to FIG. 3 and feedback loop 300. Thefeedback loop 300 generally illustrates one embodiment of how therendering module 240, the vigilance module 230, and the monitoringmodule 220 execute in parallel to render the graphical elementsaccording to the engagement level of the operator, and how theengagement level changes as the rendering module 240 renders thegraphical elements to induce responses within the operator. Thus, as theautonomous driving module 160 autonomously controls the vehicle 100according to SAE level 2 autonomy, consider that an initial engagementlevel of the operator indicates that the operator is likely not engagedand is instead distracted by infotainment or other tasks. Thus, themonitoring module 220 may indicate that a gaze of the operator isdirected away from a windshield of the vehicle 100 for an extendedperiod or repetitively toward aspects of the vehicle 100 that areunrelated to operating the vehicle 100. In response to the monitoringmodule 220 collecting the operator state information 260 about the gazeof the operator and the vigilance module 230 computing the engagementlevel that indicates this circumstance, the rendering module 240controls the AR system 180 to present one or more graphical elements.

For purposes of this discussion, consider that the rendering module 240renders a complete overlay onto a primary AR display of the AR system180 that includes graphics with a high opacity and that display sensorinformation, internal state information from the autonomous drivingmodule 160, navigation paths, object tracks, and other information ingraphical form related to a situational awareness. Thus, the AR displaytransitions from displaying, for example, relatively minimal graphicalelements and generally providing a clear/translucent view of thereal-world to being relatively highly animated with the noted elementsthat provide for an immersive experience for the operator with manyinformational elements, colors, and so on. Accordingly, the monitoringmodule 220 continues to collect information and produce the operatorstate information 260 in an updated form, while the vigilance module 230updates the engagement level to reflect a response exhibited by theoperator. Moreover, presume that the renderings provided by therendering module 240 initially induced the operator to intently view theAR display. Thus, the monitoring module 220 captures this reactionusing, for example, the cameras 126 executing an eye-tracking routine.

As such, the vigilance module 230 updates the engagement level whichthen causes the rendering module 240 to, for example, maintain thepresent graphical elements for a period of time so that the engagementlevel of the operator is sustained and does not begin to decline.However, after a defined period lapses with the operator remainingengaged, the rendering module 240 can adjust a number, variety,intensity, opacity, and/or other aspects of the graphics in order to,for example, avoid over stimulating the operator and/or to provide theoperator with a more translucent image so that the operator can developsituational awareness and vigilance about the present operatingenvironment outside of the AR system 180. However, should the engagementlevel of the operator begin to creep back toward being disengaged, thevigilance system 170 can adjust the rendering of the graphicsappropriately as identified through the feedback loop 300.

Additional aspects of mitigating vigilance decrement using augmentedreality to display dynamic graphical elements according to an engagementlevel will be discussed in relation to FIG. 4. FIG. 4 illustrates aflowchart of a method 400 that is associated with mitigating vigilancedecrement for a vehicle operator. Method 400 will be discussed from theperspective of the vigilance system 170 of FIGS. 1 and 2. While method400 is discussed in combination with the vigilance system 170, it shouldbe appreciated that the method 400 is not limited to being implementedwithin the vigilance system 170, but is instead one example of a systemthat may implement the method 400.

As an initial matter, it should be noted that the disclosedfunctionality of monitoring at 410, computing at 420, and rendering at440 and 450 generally occurs in parallel and forms a feedback loop toiteratively compute the engagement level and adjust the rendering of theat least one graphical element according to the vigilance/readinessinduced in the operator. Thus, the described method 400, in oneembodiment, includes multiple parallel processes that function tomitigate vigilance decrement and improve readiness of the operator whilethe vehicle 100 is operating according to, for example, SAE level 2autonomy or another level of autonomy during which the operator maybecome distracted. Moreover, in one embodiment, the method 400 isgenerally implemented to maintain the vigilance of the operator withrespect to supervising the autonomous operation of the vehicle 100 andreadiness of the operator to provide manual control inputs when thevehicle 100 performs a handover to the operator. In this way, the method400 can avoid difficulties associated with the vigilance decrement andotherwise disengaged operators by inducing the operator to maintain anawareness of the present operating environment and progress of thevehicle through the present operating environment.

At 410, the monitoring module 220 monitors the operator by collectingoperator state information 260. As previously mentioned, the monitoringmodule 220 acquires the data that comprises the operator stateinformation 260 from sensors of the vehicle 100 by either activelyquerying the respective sensors or by passively sniffing the data fromcommunications or electronic storage. The monitoring module 220 can thenformat or otherwise provide the collected data in an ordered manner andelectronically store the operator state information 260 in the database250 or the memory 210 for use by the vigilance module 230. In oneembodiment, the monitoring module 220 can collect the operator stateinformation from any sensors that are included within the vehicle 100.Thus, the operator state information 260, as previously discussed, cangenerally include a wide variety of information about the operator.

For example, the monitoring module 220 can control one or more cameras(e.g., cameras 126) that are disposed within an interior passengercompartment of the vehicle 100 to track eye movements of the operator.The eye movements, otherwise referred to as eye tracks, of the operatorare useful in determining whether the operator is gazing at objects infront of the vehicle 100, within the passenger compartment, and so on.Thus, the eye tracks are informative with respect to whether theoperator is actively engaged with the present operating environment ofthe vehicle. Moreover, the monitoring module 220 can use the cameras 126to track pupil dilation, rates of eye movements, and further informationabout the operator. In one embodiment, the gaze and/or eye tracks of theoperator can be a sole source of information for the operator stateinformation 260 from which the vigilance module 230 computes theengagement level. In either case, the monitoring module 220 generatesthe operator state information 260, in one embodiment, on an on-goingbasis to maintain an up-to-date characterization of the presentactivities/focus of the operator.

At 420, the vigilance module 230 computes an engagement level of theoperator. In one embodiment, the vigilance module 230 uses the operatorstate information 260 collected at 410 to characterize an extent ofvigilance decrement presently experienced by the operator. In otherwords, the engagement level is a characterization of how vigilant theoperator presently is in relation to aspects relating to supervising theautonomous operation of the vehicle 100 through the present operatingenvironment. Thus, the engagement level also characterizes thesituational awareness of the operator and the readiness of the operatorto assume control of the vehicle 100 should the autonomous drivingmodule 160 cease autonomous operation of the vehicle and execute ahandover to manual control.

Furthermore, the vigilance module 230, in one embodiment, computes theengagement level according to the vigilance model 270. As notedpreviously, the vigilance model 270 is, in one embodiment, a statisticalmodel that characterizes a likely engagement of the operator including alikely vigilance decrement of the operator as a function of indicia(e.g., the operator state information 260) about the operator. Thus, thevigilance model 270 can output a statistical likelihood as theengagement level that indicates a degree to which the operator isengaged with the present operating environment.

In further aspects, the vigilance module 230 also considers dynamicvehicle data along with the operator state information 260 to accountfor particularities of the present operating environment when computingthe engagement level. That is, the vigilance module 230 can adjust theprovided likelihood according to whether a present segment of roadway isparticularly boring, whether additional passengers are present todistract the operator, whether scenic vistas are likely to distract theoperator, whether outside temperatures are hot and the operator isuncomfortable, whether a fuel level of the vehicle is approaching empty,and so on. In general, the present disclosure envisions considering anycircumstances for which data can be ascertained and used to inform thevigilance system 170. Thus, the foregoing examples are intended as adiscussion of possible data elements and should not be construed as alimited listing.

While the vigilance model 270 is discussed along with method 400 from aheuristic perspective, in further embodiments and as previouslyindicated, the vigilance model 270 along with the vigilance module 230can implement a machine learning algorithm to learn correlations betweenthe operator state information 260, vehicle dynamics data, and othercollected data indicators in order to compute the engagement level. Ineither case, the engagement level generated at 420 informs the vigilancesystem 170 about a current state of the operator in relation toengagement with the operation of the vehicle 100.

At 430, the rendering module 240 determines whether the engagement levelsatisfies a threshold. In one embodiment, the threshold indicates aminimum engagement of the operator to be considered vigilant and readyto manually control the vehicle 100. It should be noted that while athreshold determination is discussed, in further implementations, therendering module 240 can assess the present engagement level on asliding scale or according to a further manner of classification.

For example, in one embodiment, at 430, the rendering module 240determines whether the engagement level has changed since a previousassessment. Accordingly, the rendering module 240 can compare a currentengagement level with a previous engagement level to determine apositive change (e.g., the operator is more engaged), a negative change(e.g., the operator is less engaged), or no change. In further aspects,the rendering module 240 can analyze a trend in the engagement levelover a plurality of assessments at 430. As such, depending on the trendof the engagement level, the rendering module 240 can maintain thedegree of rendering for the graphical elements as discussed further atblock 440 or adjust the rendering of the graphical elements as discussedat 450.

At 440, the rendering module 240 renders at least one graphical elementon an AR display to maintain an engagement of the operator. In oneembodiment, the rendering module 240 maintains the engagement of theoperator by continuing to render one or more elements within the ARdisplay of the AR system 180 according to the same manner as previouslyperformed. That is, if the operator is presently vigilant and engagedwith the vehicle 100, then the rendering module 240 can continuerendering the graphics in a similar manner, or, in one embodiment,reduce an intensity or other characteristic of the graphics ifpreviously intensified to gain the attention of the operator.

At 450, the rendering module 240 renders one or more graphical elementswithin the AR display to induce the operator to improve vigilance aboutthe present operation of the vehicle 100. That is, when the renderingmodule 240 determines that the engagement level for the operator is notsufficient, trending downward, or otherwise inadequate for a presentoperating environment of the vehicle 100, the rendering module 240induces behaviors/reactions within the operator by rendering variousgraphical elements on the AR display in a manner that is designed toengage the operator with the operation of the vehicle 100. Thus, invarious embodiments, the rendering module 240 can achieve the goal ofinducing the operator to engage the vehicle 100 and become more vigilantin many different ways.

Accordingly, by way of example, the rendering module 240 can render anoverlay of graphics depicting sensor data onto the AR display, rendergraphics depicting internal state information of an autonomous drivingmodule 160, render an overlay with varying opacity according to theengagement level, render a varying a number of graphic elementsaccording to the engagement level or changes to the engagement levelover time, render graphical elements within a line-of-sight of theoperator in the AR display, render text within the AR display thatincludes questions for the operator about aspects of the presentoperating environment perceived by the at least one sensor of thevehicle, and so on. Further examples and aspects of the content renderedin the AR system 180 will be discussed in relation to FIGS. 5-10subsequently.

As an additional matter, while the rendering module 240 is discussed asperforming the functionality of rendering various graphics on displaysof the AR system 180, in various embodiments, the rendering module 240can control the AR system 180 to render the graphics according tovarious electronic instructions communicated to the AR system 180 by therendering module 240.

As further illustration of how the vigilance system 170 adapts thedisplays of the AR system 180 according to the engagement level,consider FIGS. 5-10. FIG. 5 is a forward-facing view 500 from inside ofthe vehicle 100 looking through a windshield of the vehicle 100 andthrough an AR display of the AR system 180. Thus, the view 500 depictsrendered graphics produced by the rendering module 240 according to anengagement level of the operator. As shown in FIG. 5, the vigilancesystem 170 considers the engagement level of the operator to be adequateand thus renders a dashed box 510 around a truck in order to highlightthe truck within the AR display. Thus, FIG. 5 includes a singlegraphical element rendered by the rendering module 240 since theoperator is adequately engaged and, thus, aware of operating aspects ofthe vehicle 100 and the present operating environment.

By contrast, FIG. 6 illustrates an alternative view 600 of the view 500from FIG. 5. As shown in FIG. 6, the vigilance system 170 is renderingmultiple different graphics including overlays within the view 600through the AR system 180 in order to improve the vigilance of theoperator. Thus, the view 600 includes an outline box 610 to highlightthe truck that is an embellished form of the box 510 of FIG. 5.Furthermore, the view 600 is illustrated with sensor overlay 620 thatdepicts perceived data points from the LIDAR 124 rendered within the ARdisplay. Additionally, the view includes trajectory arrows 630 and 640that denote a general heading for the truck and the vehicle 100. Lastly,the view 600 depicts animated weather information 650 and 660illustrating a sun and cloud, respectively. The weather informationgenerally depicts, for example, expected weather conditions for thepresent day.

As a further example of different ways in which the vigilance system 170can render graphical elements to engage the operator, consider FIGS. 7and 8. FIG. 7 illustrates a view 700 from within the vehicle 100 that isof a roadway in front of the vehicle 100 and various dynamic and staticobjects in the present operating environment. For example, the view 700is rendered to include a path 710 of the vehicle 100 as a line on theroadway in front of the vehicle 100. Additionally, the view includes aroadway identifier 720 for the roadway onto which the vehicle 100 ismerging. As an additional note, vehicle 730 is not rendered with anyadditional graphical elements but is instead provided as a standardreal-world element without any modification through the AR system 180.

Turning to the view 800 of FIG. 8, additional elements are illustratedin the view 800 in comparison to the view 700. For example, thevigilance system 170 renders the additional elements with glowing,flashing, or more intense visual impact when the operator is not engagedwith the vehicle 100 and is instead, for example, using a personalelectronic device, performing another distracting activity, or, moregenerally, suffering from vigilance decrement. The path 810 isillustrated with more intense coloration than the path 710 from FIG. 7.Moreover, additional text 840 that describes present actions andinternal state information of the autonomous driving module 160 is alsorendered within the AR display. A route identifier 820 and a vehicle 830are illustrated with glowing colors in the view 800. As an additionalexample, the vehicle 100 uses one or more sensors to identify the routenumber and then renders a question to the operator as a graphic textelement 850 displayed floating in the sky in front of the vehicle 100.As such, the operator can respond to the question using voice commandsor another form of input at which point the vigilance system 170 canremove the text or proceed to present further questions depending onchanges to the engagement level of the operator. In this way, thevigilance system 170 adjusts how graphics are rendered in the AR displayto further engage the operator.

An additional example is illustrated in FIG. 9, which shows a similarforward-facing view from the vehicle 100. However, the view 900illustrates a volumetric scan region 910 for data provided via a radarsensor 123. The region 910 is rendered to inform the operator of wherethe radar 123 is presently scanning. Additionally, detected objects 920,930, and 940 are outlined with graphic boxes and arrows to show theyhave been detected and also to show associated trajectories of theobjects 920, 930, and 940 do not intersect with the vehicle 100.

FIG. 10 illustrates a rear-aerial view 1000 of a present operatingenvironment of the vehicle 100 that is rendered with augmented reality(AR) graphics. That is, the aerial view 1000 is not necessarily a viewthat would be presented to the operator of the vehicle 100 but isinstead provided as a representative rendering of how the vigilancesystem 170 can control the AR system 180 and how the present operatingenvironment of the vehicle 100 can be rendered around the vehicle 100depending on which direction the operator is presently gazing. Thus, asshown in FIG. 10, the present operating environment is illustrated asbeing rendered with multiple different graphical elements. In general,the view 1000 is comprised of sensor data from, for example, the Lidar124 in an overlay texture that is pervasive throughout the view 1000 andillustrates how the Lidar 124 perceives various objects and surfaces.Moreover, various tracking information for the objects is also renderedwithin the view 1000 to provide insights to the operator about internaldecision making of the autonomous driving module 160. Additionally,objects 1005-1045 are vehicles that are illustrated with outline boxesto show they have been detected by the vehicle 100 and are also shownwith graphical elements depicting information about respectivetrajectories. Furthermore, objects 1050-1090 are detected objects in theenvironment that are similarly rendered with various graphics to conveysituational information about the objects 1050-1090.

As an additional note, the view 1000 represents an example that isrelatively highly rendered with graphical elements. Thus, segments ofthe view 1000 from perspectives within the vehicle 100 may be providedto the operator when, for example, the vigilance system 170 determinesthe operator is not adequately engaged. Moreover, the vigilance system170 may render the graphic elements in a more translucent form when theoperator becomes further engaged with the vehicle 100.

FIG. 1 will now be discussed in full detail as an example vehicleenvironment within which the system and methods disclosed herein mayoperate. In some instances, the vehicle 100 is configured to switchselectively between an autonomous mode, one or more semi-autonomousoperational modes, and/or a manual mode. Such switching also referred toas handover when transitioning to a manual mode can be implemented in asuitable manner, now known or later developed. “Manual mode” means thatall of or a majority of the navigation and/or maneuvering of the vehicleis performed according to inputs received from a user (e.g., humandriver/operator).

In one or more embodiments, the vehicle 100 is an autonomous vehicle. Asused herein, “autonomous vehicle” refers to a vehicle that operates inan autonomous mode. “Autonomous mode” refers to navigating and/ormaneuvering the vehicle 100 along a travel route using one or morecomputing systems to control the vehicle 100 with minimal or no inputfrom a human driver/operator. In one or more embodiments, the vehicle100 is highly automated or completely automated. In one embodiment, thevehicle 100 is configured with one or more semi-autonomous operationalmodes in which one or more computing systems perform a portion of thenavigation and/or maneuvering of the vehicle along a travel route, and avehicle operator (i.e., driver) provides inputs to the vehicle toperform a portion of the navigation and/or maneuvering of the vehicle100 along a travel route. Thus, in one or more embodiments, the vehicle100 operates autonomously according to a particular defined level ofautonomy. For example, the vehicle 100 can operate according to theSociety of Automotive Engineers (SAE) automated vehicle classifications0-5. In one embodiment, the vehicle 100 operates according to SAE level2, which provides for the autonomous driving module 160 controlling thevehicle 100 by braking, accelerating, and steering without operatorinput but the driver/operator is to monitor the driving and be vigilantand ready to intervene with controlling the vehicle 100 if theautonomous module 160 fails to properly respond or is otherwise unableto adequately control the vehicle 100.

The vehicle 100 can include one or more processors 110. In one or morearrangements, the processor(s) 110 can be a main processor of thevehicle 100. For instance, the processor(s) 110 can be an electroniccontrol unit (ECU). The vehicle 100 can include one or more data stores115 for storing one or more types of data. The data store 115 caninclude volatile and/or non-volatile memory. Examples of suitable datastores 115 include RAM (Random Access Memory), flash memory, ROM (ReadOnly Memory), PROM (Programmable Read-Only Memory), EPROM (ErasableProgrammable Read-Only Memory), EEPROM (Electrically ErasableProgrammable Read-Only Memory), registers, magnetic disks, opticaldisks, hard drives, or any other suitable storage medium, or anycombination thereof. The data store 115 can be a component of theprocessor(s) 110, or the data store 115 can be operably connected to theprocessor(s) 110 for use thereby. The term “operably connected,” as usedthroughout this description, can include direct or indirect connections,including connections without direct physical contact.

In one or more arrangements, the one or more data stores 115 can includemap data 116. The map data 116 can include maps of one or moregeographic areas. In some instances, the map data 116 can includeinformation or data on roads, traffic control devices, road markings,structures, features, and/or landmarks in the one or more geographicareas. The map data 116 can be in any suitable form. In some instances,the map data 116 can include aerial views of an area. In some instances,the map data 116 can include ground views of an area, including360-degree ground views. The map data 116 can include measurements,dimensions, distances, and/or information for one or more items includedin the map data 116 and/or relative to other items included in the mapdata 116. The map data 116 can include a digital map with informationabout road geometry. The map data 116 can be high quality and/or highlydetailed.

In one or more arrangement, the map data 116 can include one or moreterrain maps 117. The terrain map(s) 117 can include information aboutthe ground, terrain, roads, surfaces, and/or other features of one ormore geographic areas. The terrain map(s) 117 can include elevation datain the one or more geographic areas. The map data 116 can be highquality and/or highly detailed. The terrain map(s) 117 can define one ormore ground surfaces, which can include paved roads, unpaved roads,land, and other things that define a ground surface.

In one or more arrangement, the map data 116 can include one or morestatic obstacle maps 118. The static obstacle map(s) 118 can includeinformation about one or more static obstacles located within one ormore geographic areas. A “static obstacle” is a physical object whoseposition does not change or substantially change over a period of timeand/or whose size does not change or substantially change over a periodof time. Examples of static obstacles include trees, buildings, curbs,fences, railings, medians, utility poles, statues, monuments, signs,benches, furniture, mailboxes, large rocks, hills. The static obstaclescan be objects that extend above ground level. The one or more staticobstacles included in the static obstacle map(s) 118 can have locationdata, size data, dimension data, material data, and/or other dataassociated with it. The static obstacle map(s) 118 can includemeasurements, dimensions, distances, and/or information for one or morestatic obstacles. The static obstacle map(s) 118 can be high qualityand/or highly detailed. The static obstacle map(s) 118 can be updated toreflect changes within a mapped area.

The one or more data stores 115 can include sensor data 119. In thiscontext, “sensor data” means any information about the sensors that thevehicle 100 is equipped with, including the capabilities and otherinformation about such sensors. As will be explained below, the vehicle100 can include the sensor system 120. The sensor data 119 can relate toone or more sensors of the sensor system 120. As an example, in one ormore arrangements, the sensor data 119 can include information on one ormore LIDAR sensors 124 of the sensor system 120.

In some instances, at least a portion of the map data 116 and/or thesensor data 119 can be located in one or more data stores 115 locatedonboard the vehicle 100. Alternatively, or in addition, at least aportion of the map data 116 and/or the sensor data 119 can be located inone or more data stores 115 that are located remotely from the vehicle100.

As noted above, the vehicle 100 can include the sensor system 120. Thesensor system 120 can include one or more sensors. “Sensor” means anydevice, component and/or system that can detect, and/or sense something.The one or more sensors can be configured to detect, and/or sense inreal-time. As used herein, the term “real-time” means a level ofprocessing responsiveness that a user or system senses as sufficientlyimmediate for a particular process or determination to be made, or thatenables the processor to keep up with some external process.

In arrangements in which the sensor system 120 includes a plurality ofsensors, the sensors can function independently from each other.Alternatively, two or more of the sensors can work in combination witheach other. In such a case, the two or more sensors can form a sensornetwork. The sensor system 120 and/or the one or more sensors can beoperably connected to the processor(s) 110, the data store(s) 115,and/or another element of the vehicle 100 (including any of the elementsshown in FIG. 1). The sensor system 120 can acquire data of at least aportion of the external environment of the vehicle 100 (e.g., nearbyvehicles).

The sensor system 120 can include any suitable type of sensor. Variousexamples of different types of sensors will be described herein.However, it will be understood that the embodiments are not limited tothe particular sensors described. The sensor system 120 can include oneor more vehicle sensors 121. The vehicle sensor(s) 121 can detect,determine, and/or sense information about the vehicle 100 itself. In oneor more arrangements, the vehicle sensor(s) 121 can be configured todetect, and/or sense position and orientation changes of the vehicle100, such as, for example, based on inertial acceleration. In one ormore arrangements, the vehicle sensor(s) 121 can include one or moreaccelerometers, one or more gyroscopes, an inertial measurement unit(IMU), a dead-reckoning system, a global navigation satellite system(GNSS), a global positioning system (GPS), a navigation system 147,and/or other suitable sensors. The vehicle sensor(s) 121 can beconfigured to detect, and/or sense one or more characteristics of thevehicle 100. In one or more arrangements, the vehicle sensor(s) 121 caninclude a speedometer to determine a current speed of the vehicle 100.

Alternatively, or in addition, the sensor system 120 can include one ormore environment sensors 122 configured to acquire, and/or sense drivingenvironment data. “Driving environment data” includes and data orinformation about the external environment in which an autonomousvehicle is located or one or more portions thereof. For example, the oneor more environment sensors 122 can be configured to detect, quantifyand/or sense obstacles in at least a portion of the external environmentof the vehicle 100 and/or information/data about such obstacles. Suchobstacles may be stationary objects and/or dynamic objects. The one ormore environment sensors 122 can be configured to detect, measure,quantify and/or sense other things in the external environment of thevehicle 100, such as, for example, lane markers, signs, traffic lights,traffic signs, lane lines, crosswalks, curbs proximate the vehicle 100,off-road objects, etc.

Various examples of sensors of the sensor system 120 will be describedherein. The example sensors may be part of the one or more environmentsensors 122 and/or the one or more vehicle sensors 121. Moreover, thesensor system 120 can include operator sensors that function to track orotherwise monitor aspects related to the driver/operator of the vehicle100. However, it will be understood that the embodiments are not limitedto the particular sensors described.

As an example, in one or more arrangements, the sensor system 120 caninclude one or more radar sensors 123, one or more LIDAR sensors 124,one or more sonar sensors 125, and/or one or more cameras 126. In one ormore arrangements, the one or more cameras 126 can be high dynamic range(HDR) cameras, infrared (IR) cameras and so on. In one embodiment, thecameras 126 include one or more cameras disposed within a passengercompartment of the vehicle for performing eye-tracking on theoperator/driver in order to determine a gaze of the operator/driver, aneye track of the operator/driver, and so on.

The vehicle 100 can include an input system 130. An “input system”includes any device, component, system, element or arrangement or groupsthereof that enable information/data to be entered into a machine. Theinput system 130 can receive an input from a vehicle passenger (e.g. adriver or a passenger). The vehicle 100 can include an output system135. An “output system” includes any device, component, or arrangementor groups thereof that enable information/data to be presented to avehicle passenger (e.g. a person, a vehicle passenger, etc.).

The vehicle 100 can include one or more vehicle systems 140. Variousexamples of the one or more vehicle systems 140 are shown in FIG. 1.However, the vehicle 100 can include more, fewer, or different vehiclesystems. It should be appreciated that although particular vehiclesystems are separately defined, each or any of the systems or portionsthereof may be otherwise combined or segregated via hardware and/orsoftware within the vehicle 100. The vehicle 100 can include apropulsion system 141, a braking system 142, a steering system 143,throttle system 144, a transmission system 145, a signaling system 146,and/or a navigation system 147. Each of these systems can include one ormore devices, components, and/or combination thereof, now known or laterdeveloped.

The navigation system 147 can include one or more devices, sensors,applications, and/or combinations thereof, now known or later developed,configured to determine the geographic location of the vehicle 100and/or to determine a travel route for the vehicle 100. The navigationsystem 147 can include one or more mapping applications to determine atravel route for the vehicle 100. The navigation system 147 can includea global positioning system, a local positioning system or a geolocationsystem.

The processor(s) 110, the vigilance system 170, and/or the autonomousdriving module(s) 160 can be operably connected to communicate with thevarious vehicle systems 140 and/or individual components thereof. Forexample, returning to FIG. 1, the processor(s) 110 and/or the autonomousdriving module(s) 160 can be in communication to send and/or receiveinformation from the various vehicle systems 140 to control themovement, speed, maneuvering, heading, direction, etc. of the vehicle100. The processor(s) 110, the vigilance system 170, and/or theautonomous driving module(s) 160 may control some or all of thesevehicle systems 140 and, thus, may be partially or fully autonomous.

The processor(s) 110, the vigilance system 170, and/or the autonomousdriving module(s) 160 can be operably connected to communicate with thevarious vehicle systems 140 and/or individual components thereof. Forexample, returning to FIG. 1, the processor(s) 110, the vigilance system170, and/or the autonomous driving module(s) 160 can be in communicationto send and/or receive information from the various vehicle systems 140to control the movement, speed, maneuvering, heading, direction, etc. ofthe vehicle 100. The processor(s) 110, the vigilance system 170, and/orthe autonomous driving module(s) 160 may control some or all of thesevehicle systems 140.

The processor(s) 110, the vigilance system 170, and/or the autonomousdriving module(s) 160 may be operable to control the navigation and/ormaneuvering of the vehicle 100 by controlling one or more of the vehiclesystems 140 and/or components thereof. For instance, when operating inan autonomous mode, the processor(s) 110, the vigilance system 170,and/or the autonomous driving module(s) 160 can control the directionand/or speed of the vehicle 100. The processor(s) 110, the vigilancesystem 170, and/or the autonomous driving module(s) 160 can cause thevehicle 100 to accelerate (e.g., by increasing the supply of fuelprovided to the engine), decelerate (e.g., by decreasing the supply offuel to the engine and/or by applying brakes) and/or change direction(e.g., by turning the front two wheels). As used herein, “cause” or“causing” means to make, force, compel, direct, command, instruct,and/or enable an event or action to occur or at least be in a statewhere such event or action may occur, either in a direct or indirectmanner.

The vehicle 100 can include one or more actuators 150. The actuators 150can be any element or combination of elements operable to modify, adjustand/or alter one or more of the vehicle systems 140 or componentsthereof to responsive to receiving signals or other inputs from theprocessor(s) 110 and/or the autonomous driving module(s) 160. Anysuitable actuator can be used. For instance, the one or more actuators150 can include motors, pneumatic actuators, hydraulic pistons, relays,solenoids, and/or piezoelectric actuators, just to name a fewpossibilities.

The vehicle 100 can include one or more modules, at least some of whichare described herein. The modules can be implemented ascomputer-readable program code that, when executed by a processor 110,implement one or more of the various processes described herein. One ormore of the modules can be a component of the processor(s) 110, or oneor more of the modules can be executed on and/or distributed among otherprocessing systems to which the processor(s) 110 is operably connected.The modules can include instructions (e.g., program logic) executable byone or more processor(s) 110. Alternatively, or in addition, one or moredata store 115 may contain such instructions.

In one or more arrangements, one or more of the modules described hereincan include artificial or computational intelligence elements, e.g.,neural network, fuzzy logic or other machine learning algorithms.Further, in one or more arrangements, one or more of the modules can bedistributed among a plurality of the modules described herein. In one ormore arrangements, two or more of the modules described herein can becombined into a single module.

The vehicle 100 can include one or more autonomous driving modules 160.The autonomous driving module(s) 160 can be configured to receive datafrom the sensor system 120 and/or any other type of system capable ofcapturing information relating to the vehicle 100 and/or the externalenvironment of the vehicle 100. In one or more arrangements, theautonomous driving module(s) 160 can use such data to generate one ormore driving scene models. The autonomous driving module(s) 160 candetermine position and velocity of the vehicle 100. The autonomousdriving module(s) 160 can determine the location of obstacles, or otherenvironmental features including traffic signs, trees, shrubs,neighboring vehicles, pedestrians, etc.

The autonomous driving module(s) 160 can be configured to receive,and/or determine location information for obstacles within the externalenvironment of the vehicle 100 for use by the processor(s) 110, and/orone or more of the modules described herein to estimate position andorientation of the vehicle 100, vehicle position in global coordinatesbased on signals from a plurality of satellites, or any other dataand/or signals that could be used to determine the current state of thevehicle 100 or determine the position of the vehicle 100 with respect toits environment for use in either creating a map or determining theposition of the vehicle 100 in respect to map data.

The autonomous driving module(s) 160 either independently or incombination with the vigilance system 170 can be configured to determinetravel path(s), current autonomous driving maneuvers for the vehicle100, future autonomous driving maneuvers and/or modifications to currentautonomous driving maneuvers based on data acquired by the sensor system120, driving scene models, and/or data from any other suitable source.“Driving maneuver” means one or more actions that affect the movement ofa vehicle. Examples of driving maneuvers include: accelerating,decelerating, braking, turning, moving in a lateral direction of thevehicle 100, changing travel lanes, merging into a travel lane, and/orreversing, just to name a few possibilities. The autonomous drivingmodule(s) 160 can be configured can be configured to implementdetermined driving maneuvers. The autonomous driving module(s) 160 cancause, directly or indirectly, such autonomous driving maneuvers to beimplemented. As used herein, “cause” or “causing” means to make,command, instruct, and/or enable an event or action to occur or at leastbe in a state where such event or action may occur, either in a director indirect manner. The autonomous driving module(s) 160 can beconfigured to execute various vehicle functions and/or to transmit datato, receive data from, interact with, and/or control the vehicle 100 orone or more systems thereof (e.g. one or more of vehicle systems 140).

Detailed embodiments are disclosed herein. However, it is to beunderstood that the disclosed embodiments are intended only as examples.Therefore, specific structural and functional details disclosed hereinare not to be interpreted as limiting, but merely as a basis for theclaims and as a representative basis for teaching one skilled in the artto variously employ the aspects herein in virtually any appropriatelydetailed structure. Further, the terms and phrases used herein are notintended to be limiting but rather to provide an understandabledescription of possible implementations. Various embodiments are shownin FIGS. 1-10, but the embodiments are not limited to the illustratedstructure or application.

The flowcharts and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments. In this regard, each block in the flowcharts or blockdiagrams may represent a module, segment, or portion of code, whichcomprises one or more executable instructions for implementing thespecified logical function(s). It should also be noted that, in somealternative implementations, the functions noted in the block may occurout of the order noted in the figures. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently, or theblocks may sometimes be executed in the reverse order, depending uponthe functionality involved.

The systems, components and/or processes described above can be realizedin hardware or a combination of hardware and software and can berealized in a centralized fashion in one processing system or in adistributed fashion where different elements are spread across severalinterconnected processing systems. Any kind of processing system oranother apparatus adapted for carrying out the methods described hereinis suited. A typical combination of hardware and software can be aprocessing system with computer-usable program code that, when beingloaded and executed, controls the processing system such that it carriesout the methods described herein. The systems, components and/orprocesses also can be embedded in a computer-readable storage, such as acomputer program product or other data programs storage device, readableby a machine, tangibly embodying a program of instructions executable bythe machine to perform methods and processes described herein. Theseelements also can be embedded in an application product which comprisesall the features enabling the implementation of the methods describedherein and, which when loaded in a processing system, is able to carryout these methods.

Furthermore, arrangements described herein may take the form of acomputer program product embodied in one or more computer-readable mediahaving computer-readable program code embodied, e.g., stored, thereon.Any combination of one or more computer-readable media may be utilized.The computer-readable medium may be a computer-readable signal medium ora computer-readable storage medium. The phrase “computer-readablestorage medium” means a non-transitory storage medium. Acomputer-readable storage medium may be, for example, but not limitedto, an electronic, magnetic, optical, electromagnetic, infrared, orsemiconductor system, apparatus, or device, or any suitable combinationof the foregoing. More specific examples (a non-exhaustive list) of thecomputer-readable storage medium would include the following: a portablecomputer diskette, a hard disk drive (HDD), a solid-state drive (SSD), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a portable compact disc read-only memory (CD-ROM), adigital versatile disc (DVD), an optical storage device, a magneticstorage device, or any suitable combination of the foregoing. In thecontext of this document, a computer-readable storage medium may be anytangible medium that can contain, or store a program for use by or inconnection with an instruction execution system, apparatus, or device.

Program code embodied on a computer-readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber, cable, RF, etc., or any suitable combination ofthe foregoing. Computer program code for carrying out operations foraspects of the present arrangements may be written in any combination ofone or more programming languages, including an object-orientedprogramming language such as Java™ Smalltalk, C++ or the like andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The program codemay execute entirely on the user's computer, partly on the user'scomputer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer, or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

The terms “a” and “an,” as used herein, are defined as one or more thanone. The term “plurality,” as used herein, is defined as two or morethan two. The term “another,” as used herein, is defined as at least asecond or more. The terms “including” and/or “having,” as used herein,are defined as comprising (i.e. open language). The phrase “at least oneof . . . and . . . ” as used herein refers to and encompasses any andall possible combinations of one or more of the associated listed items.As an example, the phrase “at least one of A, B, and C” includes A only,B only, C only, or any combination thereof (e.g. AB, AC, BC or ABC).

Aspects herein can be embodied in other forms without departing from thespirit or essential attributes thereof. Accordingly, reference should bemade to the following claims, rather than to the foregoingspecification, as indicating the scope hereof.

What is claimed is:
 1. A vigilance system for improving vigilance andreadiness of an operator in an autonomous vehicle that includes anaugmented reality (AR) display, comprising: one or more processors; amemory communicably coupled to the one or more processors and storing: avigilance module including instructions that when executed by the one ormore processors cause the one or more processors to compute anengagement level of the operator according to operator state informationmonitored from at least one internal sensor of the autonomous vehicle tocharacterize an extent of vigilance decrement and readiness presentlyexhibited by the operator relative to operating characteristics of theautonomous vehicle including an external environment around theautonomous vehicle and at least semi-autonomous operation of theautonomous vehicle; and a rendering module including instructions thatwhen executed by the one or more processors cause the one or moreprocessors to dynamically render, on the AR display, at least onegraphical element that is a visualization based, at least in part, onsensor data about the external environment from at least one externalsensor of the autonomous vehicle, wherein the rendering module includesinstructions to dynamically render the at least one graphical element byvarying visual characteristics of the at least one graphical element asa function of the engagement level in order to improve the engagementlevel of the operator in regards to the readiness and the vigilance ofthe operator for i) a manual handover of control to the operator by theautonomous vehicle and ii) independent intervention by the operator totakeover control of the autonomous vehicle.
 2. The vigilance system ofclaim 1, wherein the rendering module includes instructions todynamically render the at least one graphical element includinginstructions to vary visual characteristics of the at least onegraphical element based, at least in part, on changes in the engagementlevel over time, and wherein the rendering module includes instructionsto induce vigilance and readiness within the operator to improveoperation of the autonomous vehicle to operate at leastsemi-autonomously by avoiding lapses in the vigilance and the readinessof the operator and ensuring the operator is prepared to control theautonomous vehicle when the at least semi-autonomous operation of theautonomous vehicle is insufficient.
 3. The vigilance system of claim 1,further comprising: a monitoring module including instructions that whenexecuted by the one or more processors cause the one or more processorsto collect, using the at least one internal sensor of the autonomousvehicle, the operator state information perceived about the operator,wherein the vigilance module includes instructions to compute theengagement level by characterizing the operator state informationaccording to a vigilance model to indicate a measure of a present mentalawareness of the operator in relation to the operating characteristicsas a statistical likelihood, and wherein the vigilance model is astatistical model that characterizes at least a likely vigilancedecrement of the operator as a function of the operator stateinformation according to the external environment.
 4. The vigilancesystem of claim 1, wherein the rendering module includes instructions todynamically render the at least one graphical element includinginstructions to render the at least one graphical element as avisualization of internal state information that indicates presentdecisions and situational awareness of how the autonomous vehicle isperforming the at least semi-autonomous operation, and wherein therendering module includes instructions to dynamically render by varyingthe visual characteristics of the at least one graphical elementincluding changing which graphical elements are displayed and a mannerin which the graphical elements are displayed within the AR display. 5.The vigilance system of claim 4, wherein the rendering module includesinstructions to change a manner in which the graphical elements aredisplayed including instructions to modify an intensity, a style, and alocation within the AR display of the at least one graphical element tobe apparent and distinct within a line-of-sight of the operator when theengagement level indicates that the vigilance and the readiness of theoperator is diminishing.
 6. The vigilance system of claim 1, wherein therendering module includes instructions to dynamically render the atleast one graphical element including instructions to generate the atleast one graphical element as an overlay of the AR display using thesensor data about the external environment, wherein the overlay is avisualization of the sensor data as perceived by at least one externalsensor of the autonomous vehicle, and wherein the at least one externalsensor includes a Light Detection and Ranging (LiDAR) sensor.
 7. Thevigilance system of claim 1, wherein the at least semi-autonomousoperation of the autonomous vehicle is according to at least asupervised autonomy standard as electronically controlled by anautonomous driving module of the autonomous vehicle, wherein thevigilance module includes instructions to compute the engagement levelincluding instructions to analyze dynamic vehicle data along with theoperator state information to account for the external environment ofthe autonomous vehicle when computing the engagement level, and whereinthe dynamic vehicle data includes at least telematics of the autonomousvehicle, and a current location of the autonomous vehicle.
 8. Thevigilance system of claim 1, wherein the vigilance module includesinstructions to compute the engagement level including instructions touse at least gaze data from an eye-tracking camera that identifies adirection in which the operator is presently gazing to compute theengagement level.
 9. A non-transitory computer-readable medium forimproving vigilance and readiness of an operator in an autonomousvehicle that includes an augmented reality (AR) display and storinginstructions that when executed by one or more processors cause the oneor more processors to: compute an engagement level of the operatoraccording to operator state information monitored from at least oneinternal sensor of the autonomous vehicle to characterize an extent ofvigilance decrement and readiness presently exhibited by the operatorrelative to operating characteristics of the autonomous vehicleincluding an external environment around the autonomous vehicle and atleast semi-autonomous operation of the autonomous vehicle; anddynamically render, on the AR display, at least one graphical elementthat is a visualization based, at least in part, on sensor data aboutthe external environment from at least one external sensor of theautonomous vehicle, wherein the instructions to dynamically render theat least one graphical element include instructions to vary visualcharacteristics of the at least one graphical element as a function ofthe engagement level in order to improve the engagement level of theoperator in regards to the readiness and the vigilance of the operatorfor i) a manual handover of control to the operator by the autonomousvehicle and ii) independent intervention by the operator to takeovercontrol of the autonomous vehicle.
 10. The non-transitorycomputer-readable medium of claim 9, wherein the instructions todynamically render the at least one graphical element includeinstructions to vary visual characteristics of the at least onegraphical element based, at least in part, on changes in the engagementlevel over time, and wherein the instructions to induce vigilance andreadiness within the operator improve operation of the autonomousvehicle to operate at least semi-autonomously by avoiding lapses in thevigilance and the readiness of the operator and ensuring the operator isprepared to control the autonomous vehicle when the at leastsemi-autonomous operation of the autonomous vehicle is insufficient. 11.The non-transitory computer-readable medium of claim 9, furthercomprising: instructions that when executed by the one or moreprocessors cause the one or more processors to collect, using at leastone internal sensor of the autonomous vehicle, operator stateinformation perceived about the operator, wherein the instructions tocompute the engagement level include instructions to characterize theoperator state information according to a vigilance model to indicate ameasure of a present mental awareness of the operator in relation to theoperating characteristics as a statistical likelihood, and wherein thevigilance model is a statistical model that characterizes at least alikely vigilance decrement of the operator as a function of the operatorstate information according to the external environment.
 12. Thenon-transitory computer-readable medium of claim 9, wherein theinstructions to dynamically render the at least one graphical elementinclude instructions to render the at least one graphical element as avisualization of internal state information that indicates presentdecisions and situational awareness of how the autonomous vehicle isperforming the at least semi-autonomous operation, and wherein theinstructions to dynamically render include instructions to vary thevisual characteristics of the at least one graphical element includingchanging which graphical elements are displayed and a manner in whichthe graphical elements are displayed within the AR display.
 13. Thenon-transitory computer-readable medium of claim 9, wherein theinstructions to dynamically render the at least one graphical elementincludes instructions to generate the at least one graphical element asan overlay of the AR display using the sensor data about the externalenvironment, wherein the overlay is a visualization of the sensor dataas perceived by at least one external sensor of the autonomous vehicle,wherein the at least one external sensor includes a Light Detection andRanging (LiDAR) sensor, and wherein the at least semi-autonomousoperation of the autonomous vehicle is according to at least asupervised autonomy standard as electronically controlled by anautonomous driving module of the autonomous vehicle.
 14. A method ofimproving vigilance and readiness of an operator in an autonomousvehicle that includes an augmented reality (AR) display, comprising:computing, using a processor of the autonomous vehicle, an engagementlevel of the operator according to operator state information monitoredfrom at least one internal sensor of the autonomous vehicle tocharacterize an extent of vigilance decrement and readiness presentlyexhibited by the operator relative to operating characteristics of theautonomous vehicle including an external environment around theautonomous vehicle and at least semi-autonomous operation of theautonomous vehicle; and dynamically rendering, by the processor on theAR display, at least one graphical element that is a visualizationbased, at least in part, on sensor data about the external environmentfrom at least one external sensor of the autonomous vehicle, whereindynamically rendering the at least one graphical element includesvarying visual characteristics of the at least one graphical element asa function of the engagement level in order to improve the engagementlevel of the operator in regards to the readiness and the vigilance ofthe operator for i) a manual handover of control to the operator by theautonomous vehicle and ii) independent intervention by the operator totakeover control of the autonomous vehicle.
 15. The method of claim 14,wherein dynamically rendering the at least one graphical elementincludes varying visual characteristics of the at least one graphicalelement based, at least in part, on changes in the engagement level overtime, and wherein dynamically rendering the at least one graphicalelement induces vigilance and readiness within the operator improvingoperation of the autonomous vehicle to operate at leastsemi-autonomously by avoiding lapses in the vigilance and the readinessof the operator and ensuring the operator is prepared to control theautonomous vehicle when the at least semi-autonomous operation of theautonomous vehicle is insufficient.
 16. The method of claim 14, furthercomprising: collecting, using the at least one internal sensor of theautonomous vehicle, the operator state information perceived about theoperator, wherein computing the engagement level includes characterizingthe operator state information according to a vigilance model toindicate a measure of a present mental awareness of the operator inrelation to the operating characteristics as a statistical likelihood,and wherein the vigilance model is a statistical model thatcharacterizes at least a likely vigilance decrement of the operator as afunction of the operator state information according to the externalenvironment.
 17. The method of claim 14, wherein dynamically renderingthe at least one graphical element further includes rendering the atleast one graphical element as a visualization of internal stateinformation that indicates present decisions and situational awarenessof how the autonomous vehicle is performing the at least semi-autonomousoperation, and wherein dynamically rendering by varying the visualcharacteristics of the at least one graphical element includes changingwhich graphical elements are displayed and a manner in which thegraphical elements are displayed within the AR display.
 18. The methodof claim 17, wherein changing a manner in which the graphical elementsare displayed includes modifying an intensity, a style, and a locationwithin the AR display of the at least one graphical element to beapparent and distinct within a line-of-sight of the operator when theengagement level indicates that the vigilance and the readiness of theoperator is diminishing.
 19. The method of claim 14, wherein dynamicallyrendering the at least one graphical element includes generating the atleast one graphical element as an overlay of the AR display using thesensor data about the external environment, wherein the overlay is avisualization of the sensor data as perceived by the at least oneexternal sensor of the autonomous vehicle, and wherein the at least oneexternal sensor includes a Light Detection and Ranging (LiDAR) sensor.20. The method of claim 14, wherein the at least semi-autonomousoperation of the autonomous vehicle is according to at least asupervised autonomy standard, wherein computing the engagement levelfurther includes analyzing dynamic vehicle data along with the operatorstate information to account for the external environment of theautonomous vehicle when computing the engagement level, wherein thedynamic vehicle data includes at least telematics of the autonomousvehicle, and a current location of the autonomous vehicle, and whereincomputing the engagement level includes collecting at least gaze datafrom an eye-tracking camera that identifies a direction in which theoperator is presently gazing.